Repository navigation
[SSH] Fail fast when the driver proxy refuses the tunnel - #7057
Merged
Merged
Conversation
When an administrator disables port forwarding to serverless compute, the driver proxy answers every request to the tunnel server with 403 FEATURE_DISABLED. ssh connect treated that like a server that was still starting, kept polling /metadata for the full 5-minute timeout, and then failed with a generic "status code 403" error while the compute it had started stayed up. A 403 from /metadata is now final: connect stops on the first one with the proxy's own message, both in the wait loop and on the reuse path (where it would otherwise submit another server), and cancels the run it submitted. Every other non-200, such as 400 INVALID_STATE from a stale metadata.json, is still retried. The test server models the port_fwd_serverless_enabled setting so the acceptance test reaches the 403 through the setting rather than a stub. DECO-28889 Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The settings v2 backend decodes the PATCH body as a protobuf message, in
which false is the default, so {"boolean_val": {"value": false}} arrives
empty and is rejected with "must contain at least one subfield with a
non-default value" (seen on dogfood). The fake accepted it, so the
acceptance test turned the setting off with a request the real API
refuses. The fake now rejects it the same way, and the test sends the
name in the body, which is what the real API needs.
Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 14, 2026
Collaborator
Integration test reportCommit: 86d1e8b
Top 10 slowest tests (at least 2 minutes):
|
louis-johnston-db
approved these changes
Oct 15, 2026
anton-107
enabled auto-merge
October 15, 2026 12:11
rclarey
approved these changes
Oct 15, 2026
Collaborator
Integration test reportCommit: 4d4ed66
16 interesting tests: 12 flaky, 4 FAIL
Top 50 slowest tests (at least 2 minutes):
|
Collaborator
Integration test reportCommit: 4d4ed66
83 interesting tests: 83 FAIL
Top 50 slowest tests (at least 2 minutes):
|
dhakad22klx
pushed a commit
to dhakad22klx/cli
that referenced
this pull request
Oct 18, 2026
…mpute (databricks#7063) ## Changes A serverless `databricks ssh connect` (CPU or `--accelerator`) now reads the `port_fwd_serverless_enabled` workspace setting first, in ProxyCommand mode too. When an administrator has turned it off, connect stops before it creates the secret scope, uploads binaries or submits the server run: ``` Connecting to my-session... Error: port forwarding to serverless compute is disabled for this workspace (setting port_fwd_serverless_enabled), and the SSH tunnel needs it. Ask a workspace or account admin to enable it, or connect to a dedicated cluster with --cluster ``` - `checkPortForwardingToServerless` reads the setting with `WorkspaceSettingsV2.GetPublicWorkspaceSetting` and refuses only when the read succeeds and `effective_boolean_val` is `false`. - A read that fails is logged at debug level and connect continues. The driver proxy's 403 from databricks#7057 then stops it. - `libs/testserver` answers the settings GET, defaulting `port_fwd_serverless_enabled` to `true` like the real setting. ## Why Stacked on databricks#7057, PR 2 of DECO-28889. databricks#7057 makes connect stop on the driver proxy's 403, but by then it has uploaded binaries and started serverless compute. The setting is what the proxy enforces, so checking it first avoids both. The CLI can't see everything the proxy's gate (`PortForwardingToServerlessGate`) checks. The gate also needs its enforcement flag on and skips allow-listed ports, and the setting is only served through the public API in dev and staging (`public_api_stage: STAGING`; production answers `Setting port_fwd_serverless_enabled is not enabled for public API`). So a failed read is never an error, and the 403 handling stays as the backstop. The acceptance test for that backstop now stubs the settings read the way production answers it. ## Tests - Unit: `TestCheckPortForwardingToServerless` covers disabled, enabled, no effective value, and a failed read. Plus new `libs/testserver/settings_test.go` tests. - Acceptance: new `acceptance/ssh/connect-serverless-port-forwarding-disabled` checks the error and that connect sent no requests besides reads. `connect-serverless-feature-disabled` keeps covering the 403 path, with the settings read failing. All `acceptance/ssh` tests pass. - Manual, dogfood, setting off: serverless CPU, `--accelerator GPU_1xA10`, and ProxyCommand mode (`ssh -o ProxyCommand="databricks ssh connect --proxy --name …"`) all stop within 1 s with the error above. - Manual, dogfood, setting on: serverless connects and runs the remote command. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
arushiagarwal-db
pushed a commit
to arushiagarwal-db/cli
that referenced
this pull request
Oct 19, 2026
…es (databricks#7064) ## Changes Two new `SshTunnelErrorCategory` values for the port forwarding refusals added in databricks#7057 and databricks#7063: | Category | Reported when | Before | |---|---|---| | `PORT_FORWARDING_DISABLED` | The up-front read of `port_fwd_serverless_enabled` returns `false`, before any compute is started | `UNKNOWN` | | `DRIVER_PROXY_FORBIDDEN` | The driver proxy refuses a request to the SSH server with a 403, for a server that was already running or for one that this connection started | `SERVER_START_TIMEOUT` | Both acceptance tests now also check the `error_category` in the uploaded telemetry event. ## Why PR 3 of DECO-28889, after databricks#7057 and databricks#7063. Without these, a disabled setting shows up as slow server starts or as CLI bugs. Keeping the two apart shows how often the setting can't be read (production today) and the proxy's 403 has to catch it. The backend enum needs the matching values: https://github.com/databricks-eng/universe/pull/2802257. Either PR can land first; until the universe one does, ingestion drops these values. ## Tests - `acceptance/ssh/connect-serverless-port-forwarding-disabled` reports `PORT_FORWARDING_DISABLED`. - `acceptance/ssh/connect-serverless-feature-disabled` reports `DRIVER_PROXY_FORBIDDEN`. - All `acceptance/ssh` tests and `experimental/ssh/...` unit tests pass. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks ssh connectnow stops at the first 403 from the driver proxy and prints the proxy's own message, instead of polling/metadatauntil the 5-minute server start timeout. It also cancels the run it submitted, since no client can ever reach that server.getServerMetadataturns a 403 into the newerrDriverProxyForbidden, carrying the response body with its JSON quotes removed. It isn't joined witherrServerMetadata, so the first call (the reuse path) returns it as is instead of submitting a new server.cancelSSHServerRun). A server found running on the reuse path is never cancelled.400 INVALID_STATEfrom ametadata.jsonthat points at a terminated cluster.libs/testservergains a settings v2 PATCH handler, and its fake driver proxy returns the realFEATURE_DISABLEDbody whileport_fwd_serverless_enabledisfalse.Why
When an administrator turns off
port_fwd_serverless_enabled, the driver proxy answers every request to the tunnel server with403and the body"FEATURE_DISABLED: …"(a JSON string, not the usual error object). The client counted any non-200 as "server not up yet", so the user watched "Waiting for the SSH server to start..." for 5 minutes and then gotserver is not ok, status code 403, while the serverless compute stayed up.This is PR 1 of DECO-28889. PR 2 adds a check of
port_fwd_serverless_enabledbefore any compute is started, with this 403 handling as the fallback when the setting can't be read. PR 3 adds telemetry error categories for both.Tests
TestMetadataResponseErrorcovers a 403 with the real body, a non-JSON body and an empty body, plus a 400 and a 502 that are still retried.acceptance/ssh/connect-serverless-feature-disabledturns the setting off through the API, then checks the error and the recordedruns/cancel. It finishes in 1.6 s, so connect gives up on the first 403. All otheracceptance/sshtests still pass.1035981371775948endedTERMINATED / CANCELED.0907-190337-c3qelmbu) still connects and runs the remote command.This pull request and its description were written by Isaac.